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(54) Title: MULTIMEDIA ARCHITECTURE FOR INTERACTIVE ADVERTISING 



(57) Abstract 

An interactive multi-media system having a 
video server for providing a primary video program 
interleaved with secondary multimedia programming 
and a database for storing information about a viewer 
and the content of the primary video program. The 
secondary multimedia programming has a varying 
content determined by the data contained in the 
database, and as such, can be changed based upon an 
individual viewer's demographics in conjunction with 
the subject matter being displayed by the primary 
video program. Preferably, the system is accessed 
via the Internet, wherein the subscriber's house is 
connected to an Internet head end or node via a cable 
modem. 



,510 



T.V. 



- SET-TOP 
^ UNIT 



CABLE 
UOOGJ 



502' 



804' 



P.C. 



-508 



HEM) END 



6 !L 



GABU 
UODQf 



1* 



CABLE 
ROUTER 



VIDEO SERVER 



CO KTEMT AMD 
SUBSCRIBER 

B22> 




PROGRAMMING 

^DATAB ASE 



INTERNET 



5200 



? 1 


VKED 




SEVER 





SUBSCRIBER 
DATAB ASE 

54? 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States 
applications under the PCT. 



AM 


Armenia 


AT 


Austria 


AU 


Australia 


BB 


Barbados 


BE 


Belgium 


BF 


Burkina Faso 


BG 


Bulgaria 


BJ 


Benin 


BR 


Brazil 


BY 


Belarus 


CA 


Canada 


CF 


Central African Republic 


CG 


Congo 


CH 


Switzerland 


a 


Cote d'lvoire 


CM 


Cameroon 


CN 


China 


CS 


Czechoslovakia 


CZ 


Czech Republic 


DE 


Germany 


DK 


Denmark 


EE 


Estonia 


ES 


Spain 


Fl 


Finland 


FR 


France 


GA 


Gabon 



party to the PCT on the front pages 



GB 


United Kingdom 


GE 


Georgia 


GN 


Guinea 


GR 


Greece 


HU 


Hungary 


IE 


Ireland 


IT 


Italy 


JP 


Japan 


KE 


Kenya 


KG 


Kyrgystan 


KP 


Democratic People s Republic 




of Korea 


KR 


Republic of Korea 


KZ 


Kazakhstan 


U 


Liechtenstein 


LK 


Sri Lanka 


LR 


Liberia 


LT 


Lithuania 


LU 


Luxembourg 


LV 


Latvia 


MC 


Monaco 


MD 


Republic of Moldova 


MG 


Madagascar 


ML 


Mali 


MN 


Mongolia 


MR 


Mauritania 



pamphlets publishing international 



MW 


Malawi 


MX 


Mexico 


NE 


Niger 


NL 


Netherlands 


NO 


Norway 


NZ 


New Zealand 


PL 


Poland 


FT 


Portugal 


RO 


Romania 


RU 


Russian Federation 


SD 


Sudan 


SE 


Sweden 


SG 


Singapore 


SI 


Slovenia 


SK 


Slovakia 


SN 


Senegal 


sz 


Swaziland 


TD 


Chad 


TG 


Togo 


TJ 


Tajikistan 


TT 


Trinidad and Tobago 


UA 


Ukraine 


UC 


Uganda 


US 


United States of America 


UZ 


Uzbekistan 


VN 


Viet Nam 



WO 97/12486 



- 1 - 



PCT/US96/15465 



MULTIMEDIA ARCHITECTURE FOR INTERACTIVE ADVERTISING 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention is directed to a 
5 multimedia architecture for providing interactive 
video programming to an individual subscriber, for 
example to a television set, and, more particularly, 
directed to a multimedia architecture that provides 
on demand programming interleaved with secondary 
10 programming, such as advertising, to a subscriber 

wherein the content of the secondary programming is 
dependent upon information gathered about the 
viewer. 

15 Description of the Related Art 

Current interactive applications are 
typically limited to the providing of on-demand 
movies, and games, via existing cable systems and 
telephone systems. For example, Bell Atlantic is 

20 currently testing an on-demand movie system wherein 

subscribers are able to, for a fee, select a movie, 
from a list of movies. The selected movie is then 
transmitted to the subscribers television set. 

Currently, commercial messages, i.e. 

25 advertising, are delivered to a subscriber's TV frot 

centralized broadcast facilities at either a 
national or local a level. The national and local 
providers simply fill-in designated, time-based, 
gaps in a primary video program with commercials. 

30 The commercials are, at best, picked based upon a 
demographic profile of a probable cross-section of 
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viewers of the primary video programming. U.S. 
Patent No. 5,515,098 to Charles , incorporated herein 
by reference, sets forth a device and method for 
distributing commercial messages to an individually 
5 addressable subscriber terminal on a network. In 
general, a database is generated using standard 
known poling techniques having data regarding each 
subscriber to the network. This database is 
generated by, for example, mail -in responses, 

10 telephone poling or in-person surveying. After the 
database has been generated, subscribers receive 
individualized commercial messages during the normal 
gaps provided in the primary video programming. 
Charles further sets forth a method for matching the 

15 commercial messages to the appropriate viewer based 
upon the information stored in the database. 

However, to date, there has been no 
effective apparatus or method for providing 
interactive advertising or other multimedia services 

20 in conjunction with a primary video program. 

Further, there have been no provisions for utilizing 
the subscriber's television set to obtain 
statistical data regarding the subscribers. 
Additionally, there has been no effective apparatus 

25 or method for providing different commercials based 
upon the images being displayed in the primary video 
programming . 



SUMMARY OF THE INVENTION 
30 It is an object of the present invention 

to provide interactive advertising in conjunction 
with primary programming. 

It is another object of the present 
invention to provide an interactive advertising 
35 system wherein the content of advertising is 

dependent upon prior interactions with the receiver 
of the advertising. 

It is a further object of the present 
invention to provide a mult i -media architecture 
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wherein information obtained from a viewer alters 
the flow of material to the viewer. 

It is yet another object of the present 
invention to provide on-demand primary programming 
5 with interactive advertising. 

It is a further object of the present 
invention to provide a primary video stream which is 
interleaved with a secondary video stream at a time 
in the primary video stream wherein the images 
10 displayed by the primary video stream are related to 

the content of the secondary video stream. 

It is yet another object of the present 
invention to provide a primary video stream having 
commercial message breaks which are dependent upon 
15 the content of the primary video stream and the 

demographics of an individual subscriber. 

Additional objects and advantages of the 
invention will be set forth in part in the 
description which follows, and, in part, would be 
20 obvious from the description, or may be learned by 

practice of the invention. 

These and other objects of the invention 
are achieved in an interactive multimedia system 
comprising video server means for providing a 
25 primary video program interleaved with secondary 

programming, the video server means being adapted to 
receive information from a viewer during display of 
the primary video programming and the secondary 
program, and database means for storing the 
30 information about a viewer and analyzing the 

information to generate viewer data which causes the 
video server means to vary the content of the 
secondary programming displayed to the viewer based 
on the viewer data. 
35 Objects of the present invention are also 

achieved in a method of gathering data from a viewer 
comprising displaying a primary video program, 
interleaving a secondary multimedia programming with 
the display of the primary video program, presenting 



questions to the viewer during at least one of the 
primary video program and the secondary multimedia 
programming, receiving answers to the questions from 
the viewer, and varying the content of the secondary 
multimedia programming based on the answers. 

Objects of the present invention are 
further achieved in a method of displaying 
comprising displaying a primary video program having 
a varying content, generating a video stream breaks 
file correlating the content of the primary video 
program with the timing at which the content is 
displayed, interleaving secondary multimedia 
programming with the display of the primary video 
program, varying the content of the secondary 
multimedia programming based on the video file 
stream breaks file. 

Objects of the present invention are also 
achieved in an interactive multimedia system 
comprising subscriber database means for storing 
subscriber data regarding demographic and 
preferences of a subscriber, a video breaks database 
means which stores video breaks data correlating a 
content of a primary video program with a timing at 
which the content exists within the primary video 
program, and video server means for providing the 
primary video program to an individual viewer 
interleaved with secondary programming selected 
based on a correlation between the subscriber data 
and the video breaks data. 

BRIEF DESCRIPTION OF THE DRAWINGS 
These and other objects and advantages of 
the invention will become apparent and more readily 
appreciated from the following description of the 
preferred embodiments, taken in conjunction with the 
accompanying drawings of which: 

Fig. 1 is a block diagram of an 
interactive advertising system in accordance with a 
first preferred embodiment of the present invention. 
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Fig, 2 is a block diagram of a video 
server in accordance with the first preferred 
embodiment of the present invention. 

Fig- 3 is a broad band processing unit for 
5 use in the video server in accordance with the first 

preferred of the present invention. 

Fig. 4 is a block diagram of an 
interactive advertising system in accordance with a 
second preferred embodiment of the present 
invention. 

Fig. 5 is a block diagram of a set -top 
unit in accordance with the preferred embodiments of 
the present invention. 

Fig. 6 is a flow chart of the top level 
process for the video server in accordance with 
either the first or second preferred embodiments of 
the present invention. 

Fig. 7 is a flow chart of a receiving 
process in the video server in accordance with the 
preferred embodiments of the present invention. 

Fig. 8 is a flow chart of the send video 
data process used in the video server in accordance 
with the preferred embodiments of the present 
invention. 

Fig. 9 is a diagram of an MPEG stream as 
used in the preferred embodiment of the present 
invention. 

Fig. 10 is a block diagram of an MPEG 
stream selector and multiplexer as used within the 
video processor in accordance with the preferred 
embodiments of the present invention. 

Fig. 11 is a flow chart of an interleaving 
process in accordance with the preferred embodiments 
of the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Reference will now be made in detail to 
the present preferred embodiments of the present 
invention, examples of which are illustrated in the 
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accompanying drawings, wherein like reference 
numerals refer to like elements throughout. 

Interactive advertising involves 
interrupting a primary video programming stream, (or 
5 simply, the primary program) being shown to a viewer 

(or subscriber) generally comprising movies or 
educational material, at a predetermined time, with 
one or more secondary video programming streams (or 
simply, the secondary programs) , typically 
10 comprising advertising, educational or other 

commercial material. Additional facilities are 
provided to overlay textual and/or graphical 
material on the display and to receive input from 
the viewer. Thus, for example, the secondary video 
15 programming stream may be used to interactively 
request information from the viewer, such as 
demographics and shopping preferences. The 
information obtained from the viewer may be utilized 
to preselect additional secondary video programming 
20 streams for display at a future time. The timing of 

the display of the secondary program may depend upon 
the responses from the viewer and/or the content of 
the primary program. 

For example, when a viewer selects a movie 
25 to view as the primary program, he can be prompted 

to select whether he wishes to view the movie, for 
full charge without secondary programs, i.e. 
advertising, the same movie for half charge with 
some secondary programming presented during the 
30 movie, or the same movie at no charge with secondary 

programming including interactive advertising and 
questions, requiring a response from the viewer, 
presented during the movie. The nature of the 
secondary programming presented to the viewer can 
35 vary based on a number of determining factors, 
including the movie currently selected, a past 
history of the movies selected, the scene in the 
current movie at which the advertising will be 
displayed, previously collected demographic 
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information about the viewer. In the case of 
interactive advertising, the advertising can change 
based upon the viewer's response to the questions 
presented during the interactive advertising in 
5 addition to any of the previous mentioned 

determining factors. 

Thus, for example, if a viewer selects an 
action movie as the primary program, if at some 
point a truck is highlighted in a scene, secondary 
10 programming material can be interleaved which 

presents an advertisement related to a local truck 
dealer carrying a similar model of truck being shown 
in the primary programming. Alternatively, in an 
educational setting, the secondary programming can 
15 be tailored to a viewer's level of knowledge. For 

example, if a viewer has a low technical skill level 
(as determined by prior interactive sessions) , the 
primary programming can be interleaved with 
secondary programming matter explaining technical 
20 concepts presented in the primary program. 

Alternatively, if the viewer has a high level of 
technological awareness, the primary program would 
be interleaved only with minimal secondary 
programming material . 
25 Alternatively, the secondary programming 

can be an interactive presentation requesting input 
from the viewer in the above examples. For example, 
the viewer could be prompted to indicate whether, in 
fact, the viewer desires to see the secondary 
3 0 programming. Alternatively, the viewer can be 

requested to answer questions about the material 
seen in the primary program to determine the amount 
of secondary programming needed. This feature would 
be especially useful in the educational environment. 
35 Any information collected during the secondary 

programming sessions can be stored in a database and 
used in future sessions to further refine the 
content of the secondary programming. The preferred 
embodiments are described as an interactive 



advertising system, however the uses for such as 
system are not limited to advertising, but may be 
used, for example, for education and entertainment. 
Further, while the primary and secondary programs 
are described as being videos, they may also 
comprise animations, graphics, text, audio or any 
other multimedia medium. 

The primary program and secondary program 
are preferably encoded in the MPEG1 format. The 
MPEG1 format requires a throughput averaging 1.5 
megabits per second to provide a fair quality 
picture. An El or Tl wiring protocol is well 
adapted to handle such throughput. If higher 
quality video is desired, an MPEG2 encoding can be 
used. However, MPEG2 encoding requires a higher 
average throughput, typically in excess of 4 
megabits per second, and as such would require an 
advance network such as ATM or other broadband 
facilities. 

FIG. 1 is a block diagram of an 
interactive advertising system in accordance with a 
first preferred embodiment of the present invention. 
In accordance with the first preferred embodiment, 
the primary and secondary programming material is 
delivered to a subscriber's house 500 via the 
Internet 530. 

Each subscriber's house 500 is provided 
with a standard known TV 502 which is attached to a 
set -top unit 504, described hereinafter. The set- 
top unit 504 is connected to a standard cable modem 
506. An off-the-shelf P.C. 508 can be optionally 
provided The set-top unit 504 and the optional 
P.C. 508 provide the interface to the interactive 
advertising system. A cable modem 506 is typically 
connected to a head end 510 via an existing 
subscriber based cable network. The head end 510 
comprises a cable modem 512 connected to the cable 
modem 506 and the subscriber's house 500. The cable 
modem 512 is connected to a cable router 514 for 
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delivering individualized programming material to 
each subscriber's house 500. 

The head end 510 is connected to the 
Internet 530. Also connected to the Internet are a 

5 series of video servers 520a-520c. Each video 

server 520, discussed hereinafter, contains a 
content and subscriber database 522 and a 
programming database 524. The programming database 
524 stores the primary and secondary programming. 
10 The content and subscriber database 522 stores 

statistical information regarding each of the 
subscribers, including demographics, network address 
and other related information. Additionally, the 
content and subscriber database 22 stores content 

15 information regarding the primary and secondary 

programming stored in the programming database 524. 
For example, with respect to video movies, the 
content and subscriber database 522 would store 
timing and scene information regarding each movie so 

20 as to enable a correlation between a content of a 

scene and the subscriber's demographic data. 

A standard Web server 540 is also 
connected to the Internet 530. The Web server 540 
serves out Web pages in HTML format to the Internet 

25 530 in what is called the World Wide Web. The Web 

server 540 is connected to a subscriber database 542 
which is typically a table relating a subscriber's 
IP address with his cable router address. 

In use, a subscriber will access the Web 

30 server 540 using a standard Web Browser via an 

optional personal computer 508 or the set-top unit 
504, which in this configuration would preferably 
comprise an Internet ready device. Using the Web 
Browser, the subscriber selects a primary program, 

35 e.g. a movie from the Web pages displayed by the 
Web server 40, which he wishes to view on the TV 
502. The Web server 540 accesses the subscriber 
database 542 to resolve the subscriber's address. 
Thereafter, the Web server 540 directs the 
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appropriate video server 520 to begin delivery of 
the primary program. The appropriate video server 
520 can be selected based upon geographic location, 
the location of the subscriber's information, or 
even the location of the programming material . 
Thereafter, the video server 520 delivers the 
primary and secondary program material to the cable 
router 514 and the head end 510 via the Internet 
510. The cable router 514 routes the program 
material to the appropriate subscriber's house 500. 
As the set -top unit 504 provides a two way 
communication route with the video server 520, an 
interactive viewing session can be carried out, as 
discussed hereinafter. 

While the link to the set-top unit 504 
from the head end 510 has been described as a 
standard cable base subscriber system, one of 
ordinary skill in the art will recognize that an 
asymmetric digital subscriber link may also be 
utilized. Further, various additional links such as 
a Tl or an El data line may also be utilized. 

FIG. 2 is a block diagram of a video 
server 520 in accordance with the first preferred 
embodiment of the present invention. Specifically, 
FIG. 2 sets forth a video server 520 which can form 
an interactive advertising system for use in 
outputting signals to a broad band public network, 
for example, ATM, OC12, etc or a broadcast cable 
system as used in the preferred embodiment . This 
system is particularly adapted to providing 
programming over existing cable-TV networks 
utilizing standard co-axial cabling to each 
subscriber's home. 

In accordance with the preferred 
embodiments of the present invention, the video 
server 520 may be realized on a modular multimedia 
architecture which utilizes multiple application 
processing units (hereinafter APU) . United States 
Patent No. 5,029,199 and United States Patent No. 
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5,193,110, incorporated herein by references, 
describe a distributed control and storage systems 
which may be modified, in accordance with the 
present invention to produce a system for providing 
interactive advertising. In accordance with the 
first preferred embodiment of the present invention 
APUs 10 and 12 are utilized to handle voice and 
application data. Additionally, three broad band 
processing units, 14, 16 and 18 (hereinafter BPU, ) 
are utilized to handle video applications and video 
data, which are outputted to a router 50. The 
router 50 interfaces with the Internet 530 (or 
directly to a broad band network) . As shown in 
Figure 1, APU 10 and APU 12 share a SCSI disc 20. 
The APU 12 has access to a SCSI 22. Likewise BPU 
14, BPU 16 and BPU 18 each have access to SCSI discs 
24, 26 and 28, respectively which operate as the 
programming database 524 and holds video programming 
including movies and advertisement. 

The APUs 10 and 12 are connected to a 
narrowband digital switching matrix 30 via Tl lines. 
The narrowband digital switching matrix 30 is 
connected to a narrowband private switched telephone 
network (hereinafter PSTN) 46 via El lines 44. A 
master control unit controls the narrowband digital 
switching matrix 30 and the router 50. The master 
control unit 32 is connected to the APU 10, the APU 
12, the BPU 14, the BPU 16, the BPU 18 and the 
router 50 via an ethernet backbone 48. Preferably 
the master control unit 32 has an associated SCSI 
disc array 36. A database unit 38, for collecting 
and analyzing data from each of the various 
components, is also connected to the ethernet 
backbone 48. Preferably the database unit 38 is 
provided with a SCSI disk array 42. The database 
unit 38 acts as the content and subscriber database 
522 and stores various information about subscribers 
to the system including demographics, past viewing 
history, past answers to interactive questions, and 
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other information as necessary. Additionally, the 
database unit 38 can be programmed to make decisions 
regarding when to interrupt a movie with advertising 
and what advertising programming to use. However, 
the BPUs can also be programmed for this task. 

An multiple integration unit 52 
(hereinafter MIU) is connected to the narrowband 
PSTN 46 using an SMDI (simplified message desk 
interface) protocol. The MIU 52 is also connected 
to the ethernet backbone 48. The MIU 52 interfaces 
with other telephone systems, such as PBXS. A call 
control adjunct (CCA) 54 is connected to narrowband 
digital switching matrix 30 and to the narrowband 
PSTN 4 6 via the common channel system signaling 7 
protocol (hereinafter CSS 7) . The CCA 54 is also 
connected to the ethernet backbone 48. A network 
interface unit 56 (hereinafter NIU) is connected to 
the ethernet 48 and interfaces to other systems via 
FDD I (fiber distributed data interface) . 

As shown in Figure 2, the master control 
unit 32, the database unit 38, the narrowband 
switching matrix 30, the MIU 52, the CCA 54, the NIU 
56 and the router 50 are each preferably provided 
with a backup redundancy unit. 

The MCU 32 acts to assign an incoming call 
(in the case of voice mail, which can also include 
requests for video programming) , via the narrow band 
PSTN 46. The MCU32 directs requests for video 
programming, via the router 50 or the narrow band 
PSTN 46, to an appropriate APU or BPU over the 
ethernet backbone 48. Control data messages between 
the MCU 32 and the APUs or BPUs, or between the APUs 
and the BPUs, is routed over the ethernet backbone 
48. 

During operation, a request for voice or 
video services is issued via the narrowband PSTN 46 
or the Internet via the router 50. If the services 
are voice services, such as voice mail, the master 
control unit 32 routes the request to an appropriate 
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APU for service of the request by an APU via the 
narrowband digital switching matrix 30. If the 
request is for video services, the master control 32 
routes the request, via the ethernet backbone 48, to 
the appropriate BPU which provides services via the 
router 50, but may, with the addition of an 
appropriate link, be output via the in narrow band 
digital switching matrix 30. 

In general, the video programming provided 
by the BPUs may be either normal 
entertainment/educational programming or 
advertisement programming. The BPUs may access the 
database unit 38 to determine what type of 
programming to provide to a subscriber. The 
programming may be divided among the BPUs or 
arranged dynamically. Further, a single massive 
disk array may be used and the requests for services 
allocated across the BPUs based on the number of 
requests . 

FIG. 3 is a block diagram of a BPU 14 for 
use in the video server 520 in accordance with the 
first preferred embodiment of the present invention. 
The BPU 14 is provided with a CPU 64 which 
interfaces with the ethernet backbone 48 via lOmbps 
ethernet adapters 66 and 68. The CPU 64 outputs a 
signal via lOOmbps ethernet adapters 70 and 72 to 
the router 50. The BPU 14 is provided with an 
internal memory 74. An array of SCSI adapters 76 
interface with an associated array of SCSI discs 78 
which preferably store the primary and secondary 
programming material. The CPU 64 can, for example, 
comprise a 200 megahertz peutium processor. In 
addition, each BPU can, for example, comprise a 
standard PC mother board provided with the 
appropriate 10 cards. 

Although the first preferred embodiment of 
the present invention has been described with 
respect to a specific configuration, will be 
recognized at the first preferred embodiment is not 
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limited to the specific configuration. For example, 
while the present system has been described as 
outputting video signals to a router 50 for 
transmission over the Internet 53, the router 50 can 
transmit information directly to the head end 510 
via a broad band PSTN or the narrow band PSTN 46. 
However, the narrow band PSTN 46 would have to 
provide an appropriate signal throughput for 
handling the video data. 

FIG. 4 is a block diagram of an 
interactive advertising system in accordance with a 
second preferred embodiment to the present 
invention. An APU 80 and an APU 82 interface via an 
ethernet network with bridge/CSU units 88 and 90, 
which in turn interface with a digital switching 
system 96 (hereinafter DSS) . Similarly, an APU 84 
and an APU 86 interface with a DSS 96 via bridge/CSU 
units 92 and 94. The bridge/CSU units 88, 90, 92 
and 94 preferably output the via El lines. The 
output of the DSS 96 is converted back into a 
ethernet signal via CSU/bridge units 98, 100, 102 
and 104. Set-top units 504a-504d interface with the 
CSU/bridge units 98, 100, 102 and 104, respectively. 
The set-top units 504a-504d facilitate interaction 
between the APUs and the subscriber via TVs 502a- 
502d. The set -top unit 504a is further connected to 
a video conferencing unit 116, which provides video 
conferencing services via the TV 502a. The set -top 
unit 504a is connected to the TV 502a via an A/B 
switch 118, which also interfaces with the video 
conferencing unit 116. An alternative line with an 
amp 120 is provided for signal clarity. 

Figure 5 is a block diagram of a set -top 
unit 504 for use in an interactive advertising 
system in accordance with the preferred embodiments 
of the present invention. Preferably, the set-top 
unit 504 is configured around general purpose 
computer, i.e. a standard PC. A CPU 122 interfaces 
with the ethernet interface 126 for communication 
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with the APUs. An I/O unit 132 allows the viewer to 
interact with the set-top unit 504 via number of 
inputs, such as a keyboard, mouse, or an infrared 
remote control- An NTSC interface card 128 provides 
a signal to the TV 502. Additionally, an overlay 
processing unit 130 enables the CPU 122 to overlay 
text and graphics on top of a video stream. A 
memory 124 is connected to the CPU 122. 

The overlay processing unit 130 can be 
used to display texts and graphics to a viewer in 
conjunction with, or independently of the primary or 
secondary programming currently being displayed. 
For example, a list of movies could be displayed 
overtop a primary program showing a continuous 
stream of trailers showing each movie. Thereafter, 
when a viewer selects a certain movie, such as with 
an inferred remote control unit, a keyboard or a 
mouse, the set -top unit 504 requests, from an 
assigned video server 520, a secondary program 
showing a longer trailer regarding the selected 
movie while the overlay processing unit displays a 
message requesting the viewer to input whether he 
wishes to receive the movie for free, with a lot of 
advertising, for half the price with minimal 
advertising, or for full price with no advertising. 
Further, the overlay processing unit can be used 
during the advertising programming segments to take 
an order for the advertised product, request input 
from the viewer regarding the viewers responses to 
the advertising, the desirability for future 
advertising regarding the product shown during the 
advertising radio programming, or other demographic 
information. Additionally, the overlay processing 
unit 130 can be adapted to display other services, 
such as e-mail, video mail, voice mail (within an 
appropriate speech to text translator), faxes, etc. 

In use, a viewer interacts with a set-top 
unit to request video services, i.e. primary and 
secondary programming, from the BPUs. The BPUs 
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deliver video services through the router 50 and the 
Internet 530 (in the first preferred embodiment) or 
the digital switching service 96 (in the second 
preferred embodiment) to the appropriate set -top 
unit 504, which displays the output on a TV 114. 
The overlay processing unit 130 enables the display 
of menus and other messages to the subscriber 
displaying information or prompting for input* 

FIG. 6 is a flow chart of the top level 
process for the video server in accordance with 
either the first or second preferred embodiment of 
the present invention. For the sake of clarity, the 
processor will be described with respect to the 
first preferred embodiment. The process starts in 
step 1000. Thereafter, the video server 520 waits 
until a connection request is received in step 1002. 
The connection request can come via the Web server 
540 over the Internet 530 or directly from the 
subscriber's P.C. 508. Once a connection request is 
received, it is logged in step 1004, subsequently, a 
receiving process, see FIG. 7, is started in step 
1006. After the receiving process has completed the 
process returns to step 1002. 

Fig. 7 is a flow chart of a receiving 
process in the video server in accordance with the 
preferred embodiments of the present invention. The 
receiving process is started in step 2000. 
Thereafter, the video server 520 gets a message from 
the set-top unit 504 in step 2002. In step 2004, 
the message is reviewed to determine whether it 
contains a file name. If the message contains a 
file name, the requested video file is opened in 
step 2006. The process shown and described in Fig. 
7 is directed toward providing a primary video 
stream as the primary program and a secondary video 
stream as the secondary program, but other types of 
streams may be used such as simple graphics or 
audio. The team stream indicates that the data is 
sent and received as a linear, or non-linear, 
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sequence of data. The term will become more clear 
in reference to Fig. 9 which shows a typical MPEG 
stream which preferably constitutes the primary and 
secondary video streams. Thereafter, a send video 
data process is started, see Fig. 8, in step 2008. 
In step 2009, a secondary video program is 
interactively interleaved with the primary video 
programming. When the end of file of the primary 
video programming is reached, the connected is 
closed in step 210 and the receiving process is 
ended in step 2032. 

If, in step 2004, the message does not 
contain a file name, it is deemed to be a command 
requesting a change in the flow of the primary video 
program, such as rewind, fast forward, pause, stop 
or the presentation of a secondary video program. 
In step 2012, the data file of the requesting set- 
top unit 504 is opened. Thereafter in step 2014 it 
is determined whether the message requires 
additional information, such that a question needs 
to be displayed on the TV 502. If a return question 
is to be displayed, an overlay is sent to the 
overlay processing unit 130 in the set-top unit 504 
in step 2016. Thereafter, a response is received in 
step 2018 and the process ends in step 2032. If the 
response can be handled by the set-top unit 504, 
such a process is initiated. However, if the 
response needs to be handled by the video server 
520, the response is treated as another message from 
the set-top unit and another receiving process is 
started with the response being entered as the 
message. 

If, in step 2014, the message does not 
require a return question, it is determined whether 
the viewer is viewing a "must view" video in step 
2020. A "must view" video may comprise, for 
example, advertising when the viewer has indicated 
that he wishes to receive advertising in return for 
a reduced cost of viewing the primary video 
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programming. Alternatively, a "must view" video may 
also comprise, secondary video programming for which 
the advertiser has paid a premium fee to insure that 
a certain segment of the population receives it. 
Alternatively, in an educational setting, a "must 
view" video may comprise certain key points of the 
educational material which the student is required 
to view. If the viewer is viewing a "must view" 
video, the process proceeds to step 2022 where it is 
determined whether the command requests a forward 
shift in the current video stream or an entirely new 
video stream. If command does request a forward 
shift or change from the current video stream, the 
process is waited until the current "must view" 
video stream is completed in step 2024. 

Once it has been determined that the 
viewer is not viewing a "must view" video, the 
process proceeds to step 2026 wherein the requested 
video stream is opened to the requested position in 
the command. At this point, the requested video 
data is sent in a sending process, see Fig. 8. 
Thereafter, in step 2028, secondary video 
programming is interactively interleaved into the 
primary video programming. In step 2020, when the 
end of file of the primary video programming is 
reached, the set-top is updated and the connected is 
closed. Subsequently, the process ends in step 
2032. 

Fig. 8 is a flow chart of the send video 
data process used in the video server in accordance 
with the preferred embodiments of the present 
invention. The sending process starts in step 3000. 
In step 3002 data is read from the disk and sent to 
the set-top unit 504 via the Internet 530. 
Thereafter, in step 3004, it is determined whether 
the set-top unit has closed the connection. If the 
set-top unit 504 has not closed the connection, it 
is determined in step 3006 whether the end of file 
of the data being sent has been reached. If the end 
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of file has not been reached, the process continues 
to read the disk and send data in step 3002. Once 
the set-top has closed the connection in step 3004, 
or the end of file has been reached in step 3006, 
the process ends in step 3008. 

The interleaving of secondary video 
programming with the primary video programming can 
be performed at time based intervals, or based upon 
the content of the primary video programming. 
Additionally, because of the presence of a return 
communication path from the set-top unit 504, the 
secondary video programming can also be contingent 
upon commands received from the set-top unit 504. 
Additionally, it is important to note, that the 
secondary video programming can also be considered 
primary video programming to further secondary video 
programming, thus in effect allowing a hierarchical 
tree of video programming to be presented to the 

subscriber. 

In general, in order to attain 
interleaving based upon the content of the primary 
video programming, the primary video programming 
must be analyzed prior to display, and a series of 
records generated setting forth the timing and 
subject matter of the material being displayed 
during the primary video programming. For example, 
various subject matters can be defined in 
relationship to the secondary video programming and 
assigned numbers which are then stored as a record 
with the respective timings in the primary video 

programming . 

Fig. 9 is a diagram of an MPEG stream 600 
as used in the preferred embodiment of the present 
invention. An MPEG stream 600 comprises a stream 
header 602 followed by a series of data packets 604a 
to 604n. The stream header 602 includes information 
on the time at which the first data packet in the 
stream 600 is to be played. In the preferred 
embodiments, the primary and secondary programs are 
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MPEG streams and are referred to as the primary 
video stream and the secondary video stream. 

Each data packet 604 comprises a group of 
pictures (GOP) 610 and a series of frames. In 
general, there are three types of frames, a b-frame 
612 , an i-frame 614 and a p-frame 616. The i-frame 
614 comprises a full frame, while each of the fa- 
frame 612n and the p-frame 616n contain predictive 
changes to the full frame, contained in the i-frame 
614. 

Fig. 10 is a block diagram of an MPEG 
stream selector and multiplexer 700 as used within 
the video processor in accordance with the preferred 
embodiments of the present invention. Preferably, 
the MPEG stream selector and multiplexer 700 is 
realized in software in each BPU and executed by the 
CPU 64. The MPEG stream selector and multiplexer 
700 is provided with a parameter buffer 702 
(preferably in the memory 74) and a subscriber 
parameters/MPEG stream content breaks database 704 
(preferably formed in the array of SCSI disks 78) . 
The MPEG stream selector and multiplexer 70 0 
receives and multiplexes a MPEG content stream 710, 
i.e. the primary video stream, and MPEG commercial 
segments 720a-720n, i.e. the secondary video stream. 
A single MPEG output stream 730 is output by the 
MPEG stream selector and multiplexer 700. An input 
line 740 is provided to enable the BPU or set -top 
unit 504 to control the selection of the primary and 
secondary video streams in response to commands from 
the subscriber. 

Fig. 11 is a flow chart of an interleaving 
process in accordance with the preferred embodiments 
of the present invention. The process starts in 
step 4000. The header of the primary video stream 
is copied into the parameter buffer in step 4002. 
Concurrently, the header of the primary video stream 
is delivered to the output channel 730 in step 4004. 

In step 4006, the next break point in the 
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primary video stream is determined. The break point 
is the point at which the secondary video stream 
should be interleaved with the primary video stream. 
This decision is made based upon information 
contained in the subscriber parameters /MPEG stream 
content breaks database 704 or in response to a 
command from the subscriber. Typically, the 
subscriber parameter/MPEG stream content breaks 
database 704 will be generated prior to the showing 
of the primary video stream for each individual 
subscriber, by correlating the demographic data 
contained in the content and subscriber database 522 
with timing and subject information which has been 
previously generated about the primary video 
programming. While the interleaving process is 
being shown and described as being a linear process, 
it will be recognized by one of ordinary skill in 
the art that parallel processing can be used, 
wherein the break points are determined concurrently 
with the displaying of the primary video stream. 

In step 4008, the primary video stream is 
sent to the output channel. In step 4010, it is 
determined whether the end of file of the primary 
video stream has been reached. If the end of file 
has been reached, the process ends in step 4024. 

If the end of file has not been reached, 
it is determined whether a break point has been 
reached in step 4012. If a break point has not been 
reached, the process returns to step 4008 and 
continues to send primary video stream to the output 

channel 730. 

However, if, in step 4012, a break point 
has been reached, the primary video stream is 
stopped in step 4014. Subsequently, in step 4016, a 
reset code is sent to the set-top unit. Thereafter, 
in step 4018, a secondary video stream, e.g. the 
MPEG commercial segment 720a- 720b is sent to the 
output channel 730. 

Once the end of file of the secondary 
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video stream has been reached in step 4020 , a new 
stream header 602 is generated in step 4022. A copy 
of this header is stored in the parameter buffer 
702. The new header contains new timing information 
regarding the playing of the remainder of the 
primary video stream. After the new header has been 
stored, in step 4022, the process returns to step 
4000 and the next break point is determined. 

As shown and described above, the 
preferred embodiments of the present invention 
enable an individualized video programming stream to 
be transmitted to a subscriber's house. The 
individualized video programming stream comprises a 
primary program, preferably formed of a primary 
video stream and a plurality of interleaved 
secondary programs, preferably formed of secondary 
video streams. The secondary video programs are 
interleaved, based upon input received from the 
viewer, subject matter being displayed in the 
primary material, and/or demographic and personal 
data regarding the subscriber stored in a database. 
Additionally, as the system is interactive, the data 
regarding each viewer stored in the database can be 
updated in response to questions, by the overlay 
processing unit 130, presented during the 
presentation of either the primary or secondary 
material. While the system has been described with 
respect to an interactive advertising system, one of 
ordinary skill in the art will also recognize that 
it can be utilized in a variety of educational and 
entertainment scenarios . 

Although a few preferred embodiments of 
the present invention have been shown and described, 
it would be appreciated by those skilled in the art 
that changes may be made in these embodiments 
without departing from the principles and spirit of 
the invention, the scope of which is defined in the 
claims and their equivalents. 
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What is claimed is: 

1. An interactive multimedia system 

comprising: 

video server means for providing a primary 
5 video program interleaved with secondary 

programming, said video server means being adapted 
to receive information from a viewer during display 
of the primary video programming and the secondary 
programming ; and 

10 database means for storing the information 

about a viewer and analyzing the information to 
generate viewer data which causes said video server 
means to vary the content of the secondary 
programming displayed to the viewer based on the 

15 viewer data. 

2. An interactive Multimedia system, as 
set forth in claim 1, wherein the database means is 
adapted to determine the timing at which the 

20 secondary programming is displayed based upon a 

content of the primary video program. 

3. An interactive multimedia system, as 
set forth in claim 1, further comprising: 

25 interface means which receives the primary video 

program and the secondary programming from said 
video processing means and displays the video 
programming on a monitor to the viewer. 
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4. An interactive multimedia system, as 
set forth in claim 3, wherein said interface means 
is adapted to overlay the primary video program and 
secondary programming with graphics. 

5. An interactive multimedia system, as 
set forth in claim 3, wherein said interface means 
is adapted to gather information from the viewer and 
to transmit the information to said video processing 
-unit . 

6. An interactive multimedia system, as 
set forth in claim 1, wherein the primary video 
program is a video stream, 

7. An interactive multimedia system, as 
set forth in claim 6, wherein the primary video 
program is a movie. 

8. An interactive multimedia system, as 
set forth in claim 1, wherein the secondary 
programming comprises advertisement. 

9. A method of gathering data from a 
viewer comprising: 

displaying a primary video program; 

interleaving a secondary multimedia 
programming with the display of the primary video 
program; 

presenting questions to the viewer during 
at least one of the primary video program and the 
secondary multimedia programming; 

receiving answers to the questions from 
the viewer; and 

varying the content of the secondary 
multimedia programming based on the answers. 

10. A method of displaying comprising: 
displaying a primary video program having 
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a varying content; 

generating a video stream breaks file 
correlating the content of the primary video program 
with the timing at which the content is displayed; 
5 interleaving secondary multimedia 

programming with the display of the primary video 
program; 

varying the content of the secondary 
multimedia programming based on the video file 
10 stream breaks file. 

11. An interactive multimedia system 

comprising: 

subscriber database means for storing 
15 subscriber data regarding demographic and 

preferences of a subscriber; 

a video breaks database means which stores 
video breaks data correlating a content of a primary 
video program with a timing at which the content 
20 exists within the primary video program; and 

video server means for providing the 
primary video program to an individual viewer 
interleaved with secondary programming selected 
based on a correlation between the subscriber data 
and the video breaks data. 



WO 97/12486 



PCT/US96/15465 



1 /11 




FIG. I 

SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



PCT/US96/15465 



3/11 



o 



o 
m 



en 



O 



CNJ 



£Lj£ 

If* 



to 



gfci 



CD 



il 



S.S 
is 

2| 



o 



QO 



-0 



-D 



5 SUBST1TUTE SHEET (RULE 26) 



WO 97/12486 



PCT/US96/15465 




WO 97/12486 PCT/US96/15465 

5/11 




SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



6/11 



PCT/US96/15465 




LOG CONNECTION REQUEST 



j 1006 
START A RECEIVING PROCESS 



FIG. 6 



SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



7/11 



PCT/US96/15465 



( START 



,2000 



GET MESSAGE FROM 
SET-TOP UNIT 



2002 




OPEN THE REQUESTED 
VIDEO FILE 



OPEN THE CURRENT 

DATA FILE 
FOR THE REQUESTING 
SET-TOP UNIT 






I 2016 




SEND OVERLAY 






1 




RECEIVE RESPONSE 





SEND 
VIDEO DATA 



2009 



2018 



2010 



INTERACTIVELY 
INTERLEAVE 
SECONDARY VIDEO 
PROGRAMM ING 

+ 



AT EOF CLOSE 
THE CONNECTION 



2022 



OPEN THE 
REQUESTED VIDEO 
STREAM TO THE 
REQUESTED POSITION 
AND SEND VIDEO DATA 

i 



INTERACTIVELY 
INTERLEAVE 
SECONDARY VIDEO 
PROGRAMMING 

I 



2028 



WHEN EOF IS REACHED 
UPDATE SET-TOP AND 
CLOSE THE CONNECTION 



'DOES THE COMMAND 
"REQUEST A FORWARD SHIFT IN 
THE CURRENT VIDEO STREAM OR A 
NEW VIDEO STREAM? 



YES 



WAIT UNTIL CURRENT 
VIDEO STREAM IS 
COMPLETED 



2024 



-2030 



■© 



QMTjH— 2032 



FIG. 7 

SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



8/11 



PCT/US96/15465 




FIG. 8 



SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



PCT/US96/1S465 



9 /11 




SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



10/11 



PCTAJS96/15465 



710 



720b 



720c 



720a 



7 



PARAMETER 
BUFFER 



I 



702 



700 



MPEG STREAM SELECTOR 
AND 
MULTIPLEXER 



I 



730 

2. 



740 



704 



SUBSCRIBER PARAMETERS/ 
MPEG STREAM CONTENT BREAKS 



FIG. 10 



SUBSTITUTE SHEET (RULE 26) 



WO 97/12486 



PCT/US96/15465 



11/11 



( START 

i 



4000 



COPY HEADER OF 
PRIMARY VIDEO STREAM 
INTO PARAMETER BUFFER 



DELIVER HEADER OF 
PRIMARY VIDEO STREAM 
TO OUTPUT CHANNEL 



I 



.4002 



„4004 



FIG. 1 1 



DETERMINE NEXT BREAK POINT 
FOR DELIVERING SECONDARY 
VIDEO STREAM 



4006 



SEND PRIMARY 
VIDEO STREAM TO 
OUTPUT CHANNEL 



4008 




C END j 



4024 



STOP PRIMARY 

VIDEO STREAM 




.4014 



SEND RESET CODE 
TO SET-TOP UNIT 



1 



-4016 



SEND SECONDARY 
VIDEO STREAM OT 
OUTPUT CHANNEL 




.4018 



4020 



GENERATE NEW HEADER 

COPY HEADER OF 
PRIMARY VIDEO STREAM 
INTO PARAMETER BUFFER 



X 



4022 



SUBSTITUTE SHEET (RULE 26) 



INTERNATIONAL SEARCH REPORT 



A CLASSIFICATION OF SUBJECT MATTER" 

IPC 6 H94N7/16 H04N7/173 



Into .oral Applicant* No 

PCT/US 96/15465 



Accogng <o Inttnutcoil P«ttM OwficMc (tPC) of to boOi nMMQjl djsjfiaao. IPC 

B. FIELDS SEARCHED 

Minimum dooimcntmon jeardwl (d*aifie«ion lyoem foUowtd by duaficiaon tymboU) 



IPC 6 HG4N 



searched other (run cummum dceumentanon to the extent t 



l such documents are included in the fields searched 



Electronic data base consulted during the 



international search (name of data base sad, where practical, search terms used) 



I C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category ' 



Citation of 



with indication, where appropriate, of the relevant passages 



Relevant to daim No. 



p.x 



EP.A.O 656 728 (SONY CORP) 7 June 1995 

see page 3, column 3, line 3 - line 38 
see page 4, column 5, line 43 - page 6, 
column 9, line 31 
see figures 1-4,9-11 

EP,A,0 680 214 (AT & T CORP) 2 November 
1995 

see page 2, colunn 1, line 51 - column 2, 

see page 4, column 5, line 3 - page 5, 
column 8, line 19 
see figures 3-5 

-/- 



m 



Further 



: listed m the continuation of boa C. 



m 



1-3,5-8, 
10,11 



1-3,5-8, 
10,11 



Patent family members are 



* Special categories of a ted documents : 

•A' document defining die general state of the art which is not 

considered to be of particular relevance 
'E* earlier document but published on or after the international 
ahng date 

•I - document which may throw doubts on pnonty daim(s) or 
^chToted to establish the pubhcation dau of anc^her 
atafton or other special reason (as specified) 

•O* document refemng to an oral disclosure, use, exhibition or 



T later document published after the mternanonal filing date 
^pr»n^dau P at^o7m conflict with the ap^cancei but 
atedto understand the principle or theory underlying the 



-p- document published pnor to the internanonal hung date but 
later than the priority date claimed 

Date of the actual completion of the 

10 January 1997 

Name and mailing address of the ISA 

European Patent Office, P.B. 5*1 R Paientlaan 2 
NL - 2250 HV Ripwijk 
Tel ( + 31-70) J40-2040, Tx. 31 651 epo nl t 
Fax (♦ 31-70) 340-3016 



'X' document of particular relevance; the claimed «^f»° 
cannot be considered novel or cannot be ecemderedjo^ 
involve an inventive step when the document is r — * 

• Y" dcejumcnt of particular relevance; the claimed invention 
cannot be considered to invo! ve an inventt ve ^T™*?* 
document is combined with one or more c*r*rsuch docu- 
ments, such combination bang obvious to a person stalled 
m the art. 

•A' dexuroent member of the same patent family 
Date of mailing of the international search report 



28. 01. 97 



Authorized officer 



Van der Zaal, R 



Fona PCT.lS^ai 0(i 



ifcwt) (July lW2> 



page 1 of 2 



2 



INTERNATIONAL SEARCH REPORT totQ 400IIAwUcMiOBNo 

PCT/US 96/15465 


CfCononuaoon) DOCUMENTS CONSIDERED TO BE RELEVANT 


Category' 


Citation of document, with indication, where appropriate, of the relevant passages 




X 
A 


W0.A.95 15654 (ZING SYSTEMS LP) 8 June 
1995 

see page 5, line 10 - page 7, line 8 
see page 15, line 11 - page 16, line 14 
see page 17, line 10 - line 22 
see figures 1,15 

EP.A.0 536 628 (GEN INSTRUMENT CORP) 14 
April 1993 

see page 3, line 37 - page 4, line 3 
see page 5, line 27 - line 52 
see figure 2 


9-11 

1.3,5-8, 
10,11 



Form PCT ISA/210 (crnitiaiution of uaxW tbmt) (July lf92) 



page 2 of 2 



INTERNATIONAL SEARCH REPORT 

Information on patent family members 



Into >ooaJ Application No 

PCT/US 96/15465 



Patent document 
cited in tench report 



Publication 
date 



Patent family 
members) 



Publication 
date 



EP-A-0656728 



07-06-95 



JP-A- 
CN-A- 



7203420 
1115948 



04-08-95 
31-01-96 



EP-A-0680214 


02-11-95 


US-A- 
CA-A- 
JP-A- 


5532735 
2145565 
7303087 


02-07-96 
30-10-95 
, 14-11-95 


W0-A-9515654 


08-96-95 


AU-A- 
EP-A- 


1292895 
0740884 


19-06-95 
06-11-96 


EP-A-0536628 


14-04-93 


US-A- 


5231494 


27-07-93 



Form PCT/tSA 210 (f*U»u family anoax) (July IW) 



THIS PAGE BLANK (uspto) 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 



□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHD3IT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




BLACK BORDERS 



THIS PAGE BLANK (usrn» 



